Design and Evaluation of a Pipelined Distributed Information Retrieval Architecture

Web-scale search engines deal with a volume of data and queries that forces them to make use of an index partitioned across many machines. Two main methods of partitioning an index for distributed processing have been described in the literature. In document partitioning, each processor node holds the information for a subset of documents, while in term partitioning, each node holds the information for a subset of terms. Additionally, a novel architecture, pipelining, has been proposed, offering to combine the best features of both architectures. This thesis develops a careful methodology for the experimental comparison of distributed information retrieval architectures, addressing questions such as experiment scalability and query set generation. Novel methods are proposed for accumulator pruning, and for compression of accumulators for shipping between nodes in the pipelined architecture. A meticulous experimental assessment of the three distributed architectures is then undertaken. The results demonstrate that term distribution suffers a severe processing bottleneck. Pipelining resolves term distribution’s processing bottleneck, while maintaining its superior I/O characteristics. However, pipelining suffers from serious load imbalance between the nodes, fails to fully utilise the cluster’s processing capacity, and scales poorly. Document distribution, in contrast, distributes workload evenly and scales well. Load balancing through the intelligent assignment of terms to partitions is explored, but fails to fully resolve the imbalance of the pipelined architecture. Instead, the partial replication of high-workload terms is proposed, coupled with the intelligent routing of queries. These techniques resolve pipelining’s load imbalance, allowing it to marginally outperform document distribution. The partially-replicated pipelined architecture is shown to benefit from system scale. It also significantly outperforms document distribution in a memorylimited environment, suggesting that it would also outperform for larger collections relative to cluster size. However, unlike document distribution, pipelining’s average response time at low to moderate loads does not scale. The system implementor must therefore weigh the relative benefits of the two architectures.